Prediction Model Development Study Research Articles

Application of a practice-based approach in variable selection for a prediction model development study of hospital-induced delirium

BackgroundPrognostic models of hospital-induced delirium, that include potential predisposing and precipitating factors, may be used to identify vulnerable patients and inform the implementation of tailored preventive interventions. It is recommended that, in prediction model development studies, candidate predictors are selected on the basis of existing knowledge, including knowledge from clinical practice. The purpose of this article is to describe the process of identifying and operationalizing candidate predictors of hospital-induced delirium for application in a prediction model development study using a practice-based approach.MethodsThis study is part of a larger, retrospective cohort study that is developing prognostic models of hospital-induced delirium for medical-surgical older adult patients using structured data from administrative and electronic health records. First, we conducted a review of the literature to identify clinical concepts that had been used as candidate predictors in prognostic model development-and-validation studies of hospital-induced delirium. Then, we consulted a multidisciplinary task force of nine members who independently judged whether each clinical concept was associated with hospital-induced delirium. Finally, we mapped the clinical concepts to the administrative and electronic health records and operationalized our candidate predictors.ResultsIn the review of 34 studies, we identified 504 unique clinical concepts. Two-thirds of the clinical concepts (337/504) were used as candidate predictors only once. The most common clinical concepts included age (31/34), sex (29/34), and alcohol use (22/34). 96% of the clinical concepts (484/504) were judged to be associated with the development of hospital-induced delirium by at least two members of the task force. All of the task force members agreed that 47 or 9% of the 504 clinical concepts were associated with hospital-induced delirium.ConclusionsHeterogeneity among candidate predictors of hospital-induced delirium in the literature suggests a still evolving list of factors that contribute to the development of this complex phenomenon. We demonstrated a practice-based approach to variable selection for our model development study of hospital-induced delirium. Expert judgement of variables enabled us to categorize the variables based on the amount of agreement among the experts and plan for the development of different models, including an expert-model and data-driven model.

Privacy-Preserving Federated Model Predicting Bipolar Transition in Patients With Depression: Prediction Model Development Study.

Mood disorder has emerged as a serious concern for public health; in particular, bipolar disorder has a less favorable prognosis than depression. Although prompt recognition of depression conversion to bipolar disorder is needed, early prediction is challenging due to overlapping symptoms. Recently, there have been attempts to develop a prediction model by using federated learning. Federated learning in medical fields is a method for training multi-institutional machine learning models without patient-level data sharing. This study aims to develop and validate a federated, differentially private multi-institutional bipolar transition prediction model. This retrospective study enrolled patients diagnosed with the first depressive episode at 5 tertiary hospitals in South Korea. We developed models for predicting bipolar transition by using data from 17,631 patients in 4 institutions. Further, we used data from 4541 patients for external validation from 1 institution. We created standardized pipelines to extract large-scale clinical features from the 4 institutions without any code modification. Moreover, we performed feature selection in a federated environment for computational efficiency and applied differential privacy to gradient updates. Finally, we compared the federated and the 4 local models developed with each hospital's data on internal and external validation data sets. In the internal data set, 279 out of 17,631 patients showed bipolar disorder transition. In the external data set, 39 out of 4541 patients showed bipolar disorder transition. The average performance of the federated model in the internal test (area under the curve [AUC] 0.726) and external validation (AUC 0.719) data sets was higher than that of the other locally developed models (AUC 0.642-0.707 and AUC 0.642-0.699, respectively). In the federated model, classifications were driven by several predictors such as the Charlson index (low scores were associated with bipolar transition, which may be due to younger age), severe depression, anxiolytics, young age, and visiting months (the bipolar transition was associated with seasonality, especially during the spring and summer months). We developed and validated a differentially private federated model by using distributed multi-institutional psychiatric data with standardized pipelines in a real-world environment. The federated model performed better than models using local data only.

Predictors of treatment-response to caffeine combination products, acetaminophen, acetylsalicylic acid (aspirin), and nonsteroidal anti-inflammatory drugs in acute treatment of episodic migraine.

To identify predictors of acute treatment optimization for migraine with "over-the-counter" (OTC) or prescription nonsteroidal anti-inflammatory drugs (NSAIDs) as well as other widely used OTCs including acetaminophen, caffeine combination products (CCP), and acetylsalicylic acid (ASA, aspirin) among people with episodic migraine and to develop models that predict treatment response to each class of OTCs. Efficacy of acute OTC medications for migraine varies greatly. Identifying predictors of treatment response to particular classes of medication is a step toward evidence-based personalized therapy. For this prediction model development study, we used data from 2224 participants from the American Migraine Prevalence and Prevention (AMPP) study who were aged ≥18 years, met criteria for migraine, had <15 monthly headache days, and reported being on monotherapy for acute migraine attacks with one of the following classes medications: CCP (N=711), acetaminophen (N=643), ASA (N=110), and prescription or OTC NSAIDs (N=760). The primary outcome measures of treatment optimization were adequate 2-h pain freedom (2hPF) and adequate 24-h pain relief (24hPR), which were defined by responses of half the time or more to the relevant items on the Migraine Treatment Optimization Questionnaire-6. The mean (SD) age of the participants was 46.2(13.1)years, 79.4% (1765/2224) were female, 43.7% (972/2224) reported adequate 2hPF, and 46.1% (1025/2224) reported adequate 24hPR. Those taking CCP had better 2hPF and 24PR outcomes. For those taking NSAIDs, better outcomes were associated with lower average pain intensity (2hPF: odds ratio [OR] 0.89, 95% confidence interval [CI] 0.80-0.99; 24PR: OR 0.86, 95% CI 0.77-0.96), cutaneous allodynia (2hPF: OR 0.92, 95% CI 0.89-0.96; 24PR: OR 0.91, 95% CI 0.87-0.95), depressive symptoms (2hPF: OR 0.95, 95% CI 0.92-0.98; 24PR: OR 0.95, 95% CI 0.91-0.99), and Migraine Disability Assessment Scale (MIDAS) grade (2hPF: OR 0.76, 95% CI 0.64-0.90; 24PR: OR 0.79, 95% CI 0.65-0.95). Adequate 2hPF for those taking CCP was associated with male gender (OR 1.83, 95% CI 1.21-2.77), lower average pain intensity (OR 0.80, 95% CI 0.70-0.91), lower cutaneous allodynia (OR 0.94, 95% CI 0.90-0.97), and lower Migraine Symptom Severity Scale Score (MSSS; OR 0.91, 95% CI 0.86-0.97). Adequate 24hPR for those taking CCP was associated with lower average pain intensity (OR 0.85, 95% CI 0.75-0.96), lower cutaneous allodynia (OR 0.92, 95% CI 0.89-0.96), and lower MIDAS grade (OR 0.81, 95% CI 0.68-0.96). Participants who were married (OR 1.51, 95% CI 1.05-2.19), had lower average pain intensity (OR 0.79, 95% CI 0.70-0.89), lower MSSS (OR 0.93, 95% CI 0.88-0.99), less depression (OR 0.96, 95% CI 0.93-0.99), and lower MIDAS grade (OR 0.72, 95% CI 0.59-0.87) had adequate 2hPF after taking acetaminophen. Participants who were married (OR 1.50, 95% CI 1.02-2.21), had lower pain intensity (OR 0.78, 95% CI 0.69-0.88), less depression (OR 0.95, 95% CI 0.91-0.98) and lower MIDAS grade (OR 0.53, 95% CI 0.42-0.67) had higher 24hPR following use of acetaminophen. A lower MSSS was the only factor associated with higher 2hPF and 24PR after using ASA (OR 0.78, 95% CI 0.67-0.92 and OR 0.79, 95% CI 0.67-0.93). Predictive models had modest performance in identifying responders to each class of OTC. A large subgroup of people with migraine had an inadequate response to their usual acute OTC migraine treatment 2- and 24-h after dosing. These findings suggest a need to improve OTC treatment for some and to offer prescription acute medications for others. Predictive models identified several factors associated with better treatment-response in each OTC class. Selecting OTC treatment based on factors predictive of treatment optimization might improve patient outcomes.

Sensing Psychological Well-being Using Social Media Language: Prediction Model Development Study.

Positive mental health is arguably increasingly important and can be revealed, to some extent, in terms of psychological well-being (PWB). However, PWB is difficult to assess in real time on a large scale. The popularity and proliferation of social media make it possible to sense and monitor online users' PWB in a nonintrusive way, and the objective of this study is to test the effectiveness of using social media language expression as a predictor of PWB. This study aims to investigate the predictive power of social media corresponding to ground truth well-being data in a psychological way. We recruited 1427 participants. Their well-being was evaluated using 6 dimensions of PWB. Their posts on social media were collected, and 6 psychological lexicons were used to extract linguistic features. A multiobjective prediction model was then built with the extracted linguistic features as input and PWB as the output. Further, the validity of the prediction model was confirmed by evaluating the model's discriminant validity, convergent validity, and criterion validity. The reliability of the model was also confirmed by evaluating the split-half reliability. The correlation coefficients between the predicted PWB scores of social media users and the actual scores obtained using the linguistic prediction model of this study were between 0.49 and 0.54 (P<.001), which means that the model had good criterion validity. In terms of the model's structural validity, it exhibited excellent convergent validity but less than satisfactory discriminant validity. The results also suggested that our model had good split-half reliability levels for every dimension (ranging from 0.65 to 0.85; P<.001). By confirming the availability and stability of the linguistic prediction model, this study verified the predictability of social media corresponding to ground truth well-being data from the perspective of PWB. Our study has positive implications for the use of social media to predict mental health in nonprofessional settings such as self-testing or a large-scale user study.

BLOod Test Trend for cancEr Detection (BLOTTED): protocol for an observational and prediction model development study using English primary care electronic health record data

BackgroundSimple blood tests can play an important role in identifying patients for cancer investigation. The current evidence base is limited almost entirely to tests used in isolation. However, recent evidence suggests combining multiple types of blood tests and investigating trends in blood test results over time could be more useful to select patients for further cancer investigation. Such trends could increase cancer yield and reduce unnecessary referrals. We aim to explore whether trends in blood test results are more useful than symptoms or single blood test results in selecting primary care patients for cancer investigation. We aim to develop clinical prediction models that incorporate trends in blood tests to identify the risk of cancer.MethodsPrimary care electronic health record data from the English Clinical Practice Research Datalink Aurum primary care database will be accessed and linked to cancer registrations and secondary care datasets. Using a cohort study design, we will describe patterns in blood testing (aim 1) and explore associations between covariates and trends in blood tests with cancer using mixed-effects, Cox, and dynamic models (aim 2). To build the predictive models for the risk of cancer, we will use dynamic risk modelling (such as multivariate joint modelling) and machine learning, incorporating simultaneous trends in multiple blood tests, together with other covariates (aim 3). Model performance will be assessed using various performance measures, including c-statistic and calibration plots.DiscussionThese models will form decision rules to help general practitioners find patients who need a referral for further investigation of cancer. This could increase cancer yield, reduce unnecessary referrals, and give more patients the opportunity for treatment and improved outcomes.

The Application of Machine Learning in Predicting Mortality Risk in Patients With Severe Femoral Neck Fractures: Prediction Model Development Study

Background Femoral neck fracture (FNF) accounts for approximately 3.58% of all fractures in the entire body, exhibiting an increasing trend each year. According to a survey, in 1990, the total number of hip fractures in men and women worldwide was approximately 338,000 and 917,000, respectively. In China, FNFs account for 48.22% of hip fractures. Currently, many studies have been conducted on postdischarge mortality and mortality risk in patients with FNF. However, there have been no definitive studies on in-hospital mortality or its influencing factors in patients with severe FNF admitted to the intensive care unit. Objective In this paper, 3 machine learning methods were used to construct a nosocomial death prediction model for patients admitted to intensive care units to assist clinicians in early clinical decision-making. Methods A retrospective analysis was conducted using information of a patient with FNF from the Medical Information Mart for Intensive Care III. After balancing the data set using the Synthetic Minority Oversampling Technique algorithm, patients were randomly separated into a 70% training set and a 30% testing set for the development and validation, respectively, of the prediction model. Random forest, extreme gradient boosting, and backpropagation neural network prediction models were constructed with nosocomial death as the outcome. Model performance was assessed using the area under the receiver operating characteristic curve, accuracy, precision, sensitivity, and specificity. The predictive value of the models was verified in comparison to the traditional logistic model. Results A total of 366 patients with FNFs were selected, including 48 cases (13.1%) of in-hospital death. Data from 636 patients were obtained by balancing the data set with the in-hospital death group to survival group as 1:1. The 3 machine learning models exhibited high predictive accuracy, and the area under the receiver operating characteristic curve of the random forest, extreme gradient boosting, and backpropagation neural network were 0.98, 0.97, and 0.95, respectively, all with higher predictive performance than the traditional logistic regression model. Ranking the importance of the feature variables, the top 10 feature variables that were meaningful for predicting the risk of in-hospital death of patients were the Simplified Acute Physiology Score II, lactate, creatinine, gender, vitamin D, calcium, creatine kinase, creatine kinase isoenzyme, white blood cell, and age. Conclusions Death risk assessment models constructed using machine learning have positive significance for predicting the in-hospital mortality of patients with severe disease and provide a valid basis for reducing in-hospital mortality and improving patient prognosis.

Evaluating methodological quality of prognostic prediction models on patient reported outcome measurements after total hip replacement and total knee replacement surgery: a systematic review protocol

BackgroundPrediction models for poor patient-reported surgical outcomes after total hip replacement (THR) and total knee replacement (TKR) may provide a method for improving appropriate surgical care for hip and knee osteoarthritis. There are concerns about methodological issues and the risk of bias of studies producing prediction models. A critical evaluation of the methodological quality of prediction modelling studies in THR and TKR is needed to ensure their clinical usefulness. This systematic review aims to (1) evaluate and report the quality of risk stratification and prediction modelling studies that predict patient-reported outcomes after THR and TKR; (2) identify areas of methodological deficit and provide recommendations for future research; and (3) synthesise the evidence on prediction models associated with post-operative patient-reported outcomes after THR and TKR surgeries.MethodsMEDLINE, EMBASE, and CINAHL electronic databases will be searched to identify relevant studies. Title and abstract and full-text screening will be performed by two independent reviewers. We will include (1) prediction model development studies without external validation; (2) prediction model development studies with external validation of independent data; (3) external model validation studies; and (4) studies updating a previously developed prediction model. Data extraction spreadsheets will be developed based on the CHARMS checklist and TRIPOD statement and piloted on two relevant studies. Study quality and risk of bias will be assessed using the PROBAST tool. Prediction models will be summarised qualitatively. Meta-analyses on the predictive performance of included models will be conducted if appropriate. A narrative review will be used to synthesis the evidence if there are insufficient data to perform meta-analyses.DiscussionThis systematic review will evaluate the methodological quality and usefulness of prediction models for poor outcomes after THR or TKR. This information is essential to provide evidence-based healthcare for end-stage hip and knee osteoarthritis. Findings of this review will contribute to the identification of key areas for improvement in conducting prognostic research in this field and facilitate the progress in evidence-based tailored treatments for hip and knee osteoarthritis.Systematic review registrationPROSPERO registration number CRD42021271828.

Machine Learning-Derived Prenatal Predictive Risk Model to Guide Intervention and Prevent the Progression of Gestational Diabetes Mellitus to Type 2 Diabetes: Prediction Model Development Study.

BackgroundThe increasing prevalence of gestational diabetes mellitus (GDM) is concerning as women with GDM are at high risk of type 2 diabetes (T2D) later in life. The magnitude of this risk highlights the importance of early intervention to prevent the progression of GDM to T2D. Rates of postpartum screening are suboptimal, often as low as 13% in Asian countries. The lack of preventive care through structured postpartum screening in several health care systems and low public awareness are key barriers to postpartum diabetes screening.ObjectiveIn this study, we developed a machine learning model for early prediction of postpartum T2D following routine antenatal GDM screening. The early prediction of postpartum T2D during prenatal care would enable the implementation of effective strategies for diabetes prevention interventions. To our best knowledge, this is the first study that uses machine learning for postpartum T2D risk assessment in antenatal populations of Asian origin.MethodsProspective multiethnic data (Chinese, Malay, and Indian ethnicities) from 561 pregnancies in Singapore’s most deeply phenotyped mother-offspring cohort study—Growing Up in Singapore Towards healthy Outcomes—were used for predictive modeling. The feature variables included were demographics, medical or obstetric history, physical measures, lifestyle information, and GDM diagnosis. Shapley values were combined with CatBoost tree ensembles to perform feature selection. Our game theoretical approach for predictive analytics enables population subtyping and pattern discovery for data-driven precision care. The predictive models were trained using 4 machine learning algorithms: logistic regression, support vector machine, CatBoost gradient boosting, and artificial neural network. We used 5-fold stratified cross-validation to preserve the same proportion of T2D cases in each fold. Grid search pipelines were built to evaluate the best performing hyperparameters.ResultsA high performance prediction model for postpartum T2D comprising of 2 midgestation features—midpregnancy BMI after gestational weight gain and diagnosis of GDM—was developed (BMI_GDM CatBoost model: AUC=0.86, 95% CI 0.72-0.99). Prepregnancy BMI alone was inadequate in predicting postpartum T2D risk (ppBMI CatBoost model: AUC=0.62, 95% CI 0.39-0.86). A 2-hour postprandial glucose test (BMI_2hour CatBoost model: AUC=0.86, 95% CI 0.76-0.96) showed a stronger postpartum T2D risk prediction effect compared to fasting glucose test (BMI_Fasting CatBoost model: AUC=0.76, 95% CI 0.61-0.91). The BMI_GDM model was also robust when using a modified 2-point International Association of the Diabetes and Pregnancy Study Groups (IADPSG) 2018 criteria for GDM diagnosis (BMI_GDM2 CatBoost model: AUC=0.84, 95% CI 0.72-0.97). Total gestational weight gain was inversely associated with postpartum T2D outcome, independent of prepregnancy BMI and diagnosis of GDM (P=.02; OR 0.88, 95% CI 0.79-0.98).ConclusionsMidgestation weight gain effects, combined with the metabolic derangements underlying GDM during pregnancy, signal future T2D risk in Singaporean women. Further studies will be required to examine the influence of metabolic adaptations in pregnancy on postpartum maternal metabolic health outcomes. The state-of-the-art machine learning model can be leveraged as a rapid risk stratification tool during prenatal care.Trial RegistrationClinicalTrials.gov NCT01174875; https://clinicaltrials.gov/ct2/show/NCT01174875

Predictive models for determining treatment response to nonprescription acute medications in migraine: Results from the American Migraine Prevalence and Prevention Study.

To identify predictors of acute treatment response for nonprescription (over-the-counter [OTC]) medications among people with migraine and develop improved models for predicting treatment response. Pain freedom and sustained pain relief are important priorities in the acute treatment of migraine. OTC medications are widely used for migraine; however, it is not clear which treatment works best for each patient without going through the trial and error process. A prediction model development study was completed using the 2006 American Migraine Prevalence and Prevention Study survey, from participants who were aged ≥18, met criteria and headache day frequency for episodic migraine, did not take prescription medication for migraine, and used ≥1 of the following acute migraine medication classes: acetaminophen, aspirin, NSAIDs, or caffeine containing combination products (CCP). Two items from the Migraine Treatment Optimization Questionnaire were used to evaluate treatment response, adequate 2-h pain freedom (2hPF) and 24-h pain relief (24hPR), which were defined by a response to treatment ≥half the time at 2 h and 24 h post treatment, respectively. We identified predictors of adequate treatment response and developed models to predict probability of treatment response to each medication class. The sample included 3852 participants (3038 [79.0%] females) with an average age of 45.0years (SD=12.8). Only 1602/3852 (41.6%) and 1718/3852 (44.6%) of the participants reported adequate 2hPF and 24hPR, respectively. Adequate treatment-response was significantly predicted by lower average headache pain intensity, less cutaneous allodynia, and lower depressive symptom scores. Lower migraine symptom severity was predictive of adequate 2hPF and fewer monthly headache days was predictive of adequate 24hPR. Among participants reporting OTC monotherapy (n=2168, 56.3%) individuals taking CCP were more likely to have adequate 2hPF (OR=1.55, 95% CI 1.23-1.95) and 24hPR (OR=1.79, 95% CI 1.18-1.88) in comparison with those taking acetaminophen. Predictive models were modestly predictive of responders to OTC medications (c-statistics=0.65; 95% CI 0.62-0.68). These results show that response to acute migraine treatments is not optimized in the majority of people with migraine treating with OTC medications. Predictive models can improve our ability to choose the best therapeutic option for individuals with episodic migraine and increase the proportion of patients with optimized response to treatments.

Use of Deep Learning to Predict Acute Kidney Injury After Intravenous Contrast Media Administration: Prediction Model Development Study.

BackgroundPrecise prediction of contrast media–induced acute kidney injury (CIAKI) is an important issue because of its relationship with poor outcomes.ObjectiveHerein, we examined whether a deep learning algorithm could predict the risk of intravenous CIAKI better than other machine learning and logistic regression models in patients undergoing computed tomography (CT).MethodsA total of 14,185 patients who were administered intravenous contrast media for CT at the preventive and monitoring facility in Seoul National University Hospital were reviewed. CIAKI was defined as an increase in serum creatinine of ≥0.3 mg/dL within 2 days or ≥50% within 7 days. Using both time-varying and time-invariant features, machine learning models, such as the recurrent neural network (RNN), light gradient boosting machine (LGM), extreme gradient boosting machine (XGB), random forest (RF), decision tree (DT), support vector machine (SVM), κ-nearest neighbors, and logistic regression, were developed using a training set, and their performance was compared using the area under the receiver operating characteristic curve (AUROC) in a test set.ResultsCIAKI developed in 261 cases (1.8%). The RNN model had the highest AUROC of 0.755 (0.708-0.802) for predicting CIAKI, which was superior to that obtained from other machine learning models. Although CIAKI was defined as an increase in serum creatinine of ≥0.5 mg/dL or ≥25% within 3 days, the highest performance was achieved in the RNN model with an AUROC of 0.716 (95% confidence interval [CI] 0.664-0.768). In feature ranking analysis, the albumin level was the most highly contributing factor to RNN performance, followed by time-varying kidney function.ConclusionsApplication of a deep learning algorithm improves the predictability of intravenous CIAKI after CT, representing a basis for future clinical alarming and preventive systems.

Machine Learning Approach to Predict the Probability of Recurrence of Renal Cell Carcinoma After Surgery: Prediction Model Development Study.

BackgroundRenal cell carcinoma (RCC) has a high recurrence rate of 20% to 30% after nephrectomy for clinically localized disease, and more than 40% of patients eventually die of the disease, making regular monitoring and constant management of utmost importance.ObjectiveThe objective of this study was to develop an algorithm that predicts the probability of recurrence of RCC within 5 and 10 years of surgery.MethodsData from 6849 Korean patients with RCC were collected from eight tertiary care hospitals listed in the KOrean Renal Cell Carcinoma (KORCC) web-based database. To predict RCC recurrence, analytical data from 2814 patients were extracted from the database. Eight machine learning algorithms were used to predict the probability of RCC recurrence, and the results were compared.ResultsWithin 5 years of surgery, the highest area under the receiver operating characteristic curve (AUROC) was obtained from the naïve Bayes (NB) model, with a value of 0.836. Within 10 years of surgery, the highest AUROC was obtained from the NB model, with a value of 0.784.ConclusionsAn algorithm was developed that predicts the probability of RCC recurrence within 5 and 10 years using the KORCC database, a large-scale RCC cohort in Korea. It is expected that the developed algorithm will help clinicians manage prognosis and establish customized treatment strategies for patients with RCC after surgery.

Using Automated Machine Learning to Predict the Mortality of Patients With COVID-19: Prediction Model Development Study.

BackgroundDuring a pandemic, it is important for clinicians to stratify patients and decide who receives limited medical resources. Machine learning models have been proposed to accurately predict COVID-19 disease severity. Previous studies have typically tested only one machine learning algorithm and limited performance evaluation to area under the curve analysis. To obtain the best results possible, it may be important to test different machine learning algorithms to find the best prediction model.ObjectiveIn this study, we aimed to use automated machine learning (autoML) to train various machine learning algorithms. We selected the model that best predicted patients’ chances of surviving a SARS-CoV-2 infection. In addition, we identified which variables (ie, vital signs, biomarkers, comorbidities, etc) were the most influential in generating an accurate model.MethodsData were retrospectively collected from all patients who tested positive for COVID-19 at our institution between March 1 and July 3, 2020. We collected 48 variables from each patient within 36 hours before or after the index time (ie, real-time polymerase chain reaction positivity). Patients were followed for 30 days or until death. Patients’ data were used to build 20 machine learning models with various algorithms via autoML. The performance of machine learning models was measured by analyzing the area under the precision-recall curve (AUPCR). Subsequently, we established model interpretability via Shapley additive explanation and partial dependence plots to identify and rank variables that drove model predictions. Afterward, we conducted dimensionality reduction to extract the 10 most influential variables. AutoML models were retrained by only using these 10 variables, and the output models were evaluated against the model that used 48 variables.ResultsData from 4313 patients were used to develop the models. The best model that was generated by using autoML and 48 variables was the stacked ensemble model (AUPRC=0.807). The two best independent models were the gradient boost machine and extreme gradient boost models, which had an AUPRC of 0.803 and 0.793, respectively. The deep learning model (AUPRC=0.73) was substantially inferior to the other models. The 10 most influential variables for generating high-performing models were systolic and diastolic blood pressure, age, pulse oximetry level, blood urea nitrogen level, lactate dehydrogenase level, D-dimer level, troponin level, respiratory rate, and Charlson comorbidity score. After the autoML models were retrained with these 10 variables, the stacked ensemble model still had the best performance (AUPRC=0.791).ConclusionsWe used autoML to develop high-performing models that predicted the survival of patients with COVID-19. In addition, we identified important variables that correlated with mortality. This is proof of concept that autoML is an efficient, effective, and informative method for generating machine learning–based clinical decision support tools.

Institution-Specific Machine Learning Models for Prehospital Assessment to Predict Hospital Admission: Prediction Model Development Study.

BackgroundAlthough multiple prediction models have been developed to predict hospital admission to emergency departments (EDs) to address overcrowding and patient safety, only a few studies have examined prediction models for prehospital use. Development of institution-specific prediction models is feasible in this age of data science, provided that predictor-related information is readily collectable.ObjectiveWe aimed to develop a hospital admission prediction model based on patient information that is commonly available during ambulance transport before hospitalization.MethodsPatients transported by ambulance to our ED from April 2018 through March 2019 were enrolled. Candidate predictors were age, sex, chief complaint, vital signs, and patient medical history, all of which were recorded by emergency medical teams during ambulance transport. Patients were divided into two cohorts for derivation (3601/5145, 70.0%) and validation (1544/5145, 30.0%). For statistical models, logistic regression, logistic lasso, random forest, and gradient boosting machine were used. Prediction models were developed in the derivation cohort. Model performance was assessed by area under the receiver operating characteristic curve (AUROC) and association measures in the validation cohort.ResultsOf 5145 patients transported by ambulance, including deaths in the ED and hospital transfers, 2699 (52.5%) required hospital admission. Prediction performance was higher with the addition of predictive factors, attaining the best performance with an AUROC of 0.818 (95% CI 0.792-0.839) with a machine learning model and predictive factors of age, sex, chief complaint, and vital signs. Sensitivity and specificity of this model were 0.744 (95% CI 0.716-0.773) and 0.745 (95% CI 0.709-0.776), respectively.ConclusionsFor patients transferred to EDs, we developed a well-performing hospital admission prediction model based on routinely collected prehospital information including chief complaints.

A systematic review of methodological quality of model development studies predicting prognostic outcome for resectable pancreatic cancer

ObjectivesTo assess the methodological quality of prognostic model development studies pertaining to post resection prognosis of pancreatic ductal adenocarcinoma (PDAC).Design/settingA narrative systematic review of international peer reviewed journalsData sourceSearches were...

Nomograms to predict naming decline after temporal lobe surgery in adults with epilepsy.

To develop and externally validate models to predict the probability of postoperative naming decline in adults following temporal lobe epilepsy surgery using easily accessible preoperative clinical predictors. In this retrospective, prediction model development study, multivariable models were developed in a cohort of 719 patients who underwent temporal lobe epilepsy surgery at Cleveland Clinic and externally validated in a cohort of 138 patients who underwent temporal lobe surgery at one of 3 epilepsy surgery centers in the United States (Columbia University Medical Center, Emory University School of Medicine, University of Washington School of Medicine). The development cohort was 54% female with an average age at surgery of 36 years (SD 12). Twenty-six percent of this cohort experienced clinically relevant postoperative naming decline. The model included 5 variables: side of surgery, age at epilepsy onset, age at surgery, sex, and education. When applied to the external validation cohort, the model performed very well, with excellent calibration and a c statistic (reflecting discriminatory ability) of 0.81. A second model predicting moderate to severe postoperative naming decline included 3 variables: side of surgery, age at epilepsy onset, and preoperative naming score. This model generated a c statistic of 0.84 in the external validation cohort and showed good calibration. Externally validated nomograms are provided in 2 easy-to-use formats (paper version and online calculator) clinicians can use to estimate the probability of naming decline in patients considering epilepsy surgery for treatment of pharmacoresistant temporal lobe epilepsy.

Sample size for binary logistic prediction models: Beyond events per variable criteria.

Binary logistic regression is one of the most frequently applied statistical approaches for developing clinical prediction models. Developers of such models often rely on an Events Per Variable criterion (EPV), notably EPV ≥10, to determine the minimal sample size required and the maximum number of candidate predictors that can be examined. We present an extensive simulation study in which we studied the influence of EPV, events fraction, number of candidate predictors, the correlations and distributions of candidate predictor variables, area under the ROC curve, and predictor effects on out-of-sample predictive performance of prediction models. The out-of-sample performance (calibration, discrimination and probability prediction error) of developed prediction models was studied before and after regression shrinkage and variable selection. The results indicate that EPV does not have a strong relation with metrics of predictive performance, and is not an appropriate criterion for (binary) prediction model development studies. We show that out-of-sample predictive performance can better be approximated by considering the number of predictors, the total sample size and the events fraction. We propose that the development of new sample size criteria for prediction models should be based on these three parameters, and provide suggestions for improving sample size determination.

Recognition of Atypical Symptoms of Acute Myocardial Infarction: Development and Validation of a Risk Scoring System.

Atypical symptom presentation in patients with acute myocardial infarction (AMI) is associated with longer delay in care seeking and poorer prognosis. Symptom recognition in these patients is a challenging task. Our purpose in this risk prediction model development study was to develop and validate a risk scoring system for estimating cumulative risk for atypical AMI presentation. A consecutive sample was recruited for the developmental (n = 300) and validation (n = 97) cohorts. Symptom experience was measured with the validated Chinese version of the Symptoms of Acute Coronary Syndromes Inventory. Potential predictors were identified from the literature. Multivariable logistic regression was performed to identify significant predictors. A risk scoring system was then constructed by assigning weights to each significant predictor according to their b coefficients. Five independent predictors for atypical symptom presentation were older age (≥75 years), female gender, diabetes mellitus, history of AMI, and absence of hyperlipidemia. The Hosmer and Lemeshow test (χ6 = 4.47, P = .62) indicated that this predictive model was adequate to predict the outcome. Acceptable discrimination was demonstrated, with area under the receiver operating characteristic curve as 0.74 (95% confidence interval, 0.67-0.82) (P < .001). The predictive power of this risk scoring system was confirmed in the validation cohort. Atypical AMI presentation is common. A simple risk scoring system developed on the basis of the 5 identified predictors can raise awareness of atypical AMI presentation and promote symptom recognition by estimating the cumulative risk for an individual to present with atypical AMI symptoms.

Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use.

A fundamental part of medical research is the development and validation of diagnostic and prognostic prediction models [1,2]. These prediction models aim to predict the absolute probability that a certain disease or condition is currently present (diagnostic models) or that an outcome will occur within a specific follow-up period (prognostic models) for an individual subject. Prediction models typically rely on multiple predictors, which can include demographic characteristics, medical history and physical examination items, or more complex measurements from, for example, medical imaging, electrophysiology, pathology, and biomarkers. Also for diagnostic models, estimates of probabilities are rarely based on a single test, and doctors naturally integrate several patient characteristics and symptoms [3]. A broad range of prediction modeling techniques exist, like regression approaches, neural network models, decision tree models, genetic programming models, and support vector machine learning models, although prediction models developed by a multivariable regression approach are by far prevailing. It is widely recommended that a developed prediction model should not be used in practice before being externally validated—at least once—in other individuals than those used for model development [4–7]. Unfortunately, most prediction models are poorly or not at all validated, rendering interpretation of their generalizability difficult. In addition, many systematic reviews showed that for the same outcome or same target population, numerous competing models exist [8–10]. Generally speaking, researchers often ignore existing prediction models and develop yet another prediction model from their own data [2]. This practice sustains a cycle of underpowered prediction model development studies and poor knowledge about the generalizability and applicability of developed prediction models. Evidence synthesis and meta-analysis of individual participant data (IPD) from multiple studies seems to be a unique opportunity to address these problems, as it allows researchers to develop and directly validate models on large datasets and across a wide range of populations and settings, to directly test a model’s generalizability (Fig 1) [11–13]. Fig 1 Trends in publications of IPD-MA studies focusing on the development and/or validation of diagnostic or prognostic prediction models. There is currently little guidance on how to conduct an IPD meta-analysis (IPD-MA) for developing and/or validating diagnostic or prognostic prediction models [15]. To date, most IPD-MA articles focus on estimating relative quantities, like a risk ratio, hazard ratio, or odds ratio for a specific treatment or a specific etiologic factor. In contrast, prediction modeling research is focused on developing and validating multivariable models aimed at calculating an absolute risk estimate of the combined variables, rather than estimating the relative effect of a specific treatment or etiologic factor. Furthermore, prediction modeling studies focus entirely on the role and joint contribution of multiple covariates, whereas intervention studies in principle rely on randomization to reduce the role of covariates (Table 1). Hence, IPD-MAs of randomized intervention and etiological studies, which are beyond the scope of this paper and are instead addressed in the accompanying paper [16], differ from IPD-MAs of multivariable prediction models, which are the focus of this paper. Table 1 The main differences between IPD-MA of treatment intervention studies and of multivariable prediction modeling studies. We provide an overview of the advantages and limitations of IPD-MAs aiming to develop a novel prediction model or to validate one or more existing models across multiple datasets. This overview is based on published guidelines and existing recommendations for the conduct of prediction modeling studies and of IPD-MA research. We illustrate this overview with examples of recently published IPD-MAs of prediction models across various medical domains. Our aim is to help researchers, readers, reviewers, and editors to identify and understand the key issues involved with such IPD-MA projects.

Reporting and Methods in Clinical Prediction Research: A Systematic Review

We investigated the reporting and methods of prediction studies, focusing on aims, designs, participant selection, outcomes, predictors, statistical power, statistical methods, and predictive performance measures. We used a full hand search to identify all prediction studies published in 2008 in six high impact general medical journals. We developed a comprehensive item list to systematically score conduct and reporting of the studies, based on recent recommendations for prediction research. Two reviewers independently scored the studies. We retrieved 71 papers for full text review: 51 were predictor finding studies, 14 were prediction model development studies, three addressed an external validation of a previously developed model, and three reported on a model's impact on participant outcome. Study design was unclear in 15% of studies, and a prospective cohort was used in most studies (60%). Descriptions of the participants and definitions of predictor and outcome were generally good. Despite many recommendations against doing so, continuous predictors were often dichotomized (32% of studies). The number of events per predictor as a measure of statistical power could not be determined in 67% of the studies; of the remainder, 53% had fewer than the commonly recommended value of ten events per predictor. Methods for a priori selection of candidate predictors were described in most studies (68%). A substantial number of studies relied on a p-value cut-off of p<0.05 to select predictors in the multivariable analyses (29%). Predictive model performance measures, i.e., calibration and discrimination, were reported in 12% and 27% of studies, respectively. The majority of prediction studies in high impact journals do not follow current methodological recommendations, limiting their reliability and applicability.